21/04/2024 - 27/04/2024

23/04/2024 11:20

Ran at ~1.8kHz for 30 minutes with external trigger. First time I tried I got a CCC run abort immediately. I simply restarted the frontends:
addd127296063bf0e542070ddde7d869.png

23/04/2024 11:45

Some timing info for the master "read_trigger_event":

read_trigger_event : Trigger info:
    trigger number 24571
    trigger mask 0
    time_s : 1713886819
    time_us : -2067971001
    time_done_s : 0
    time_done_us : 0
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 39 microseconds
Block 5 (Updating odb monitor) time: 4 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
read_trigger_event : Trigger info:
    trigger number 24572
    trigger mask 0
    time_s : 1713886819
    time_us : -2065393376
    time_done_s : 0
    time_done_us : 0
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 37 microseconds
Block 5 (Updating odb monitor) time: 4 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
read_trigger_event : Trigger info:
    trigger number 24571
    trigger mask 0
    time_s : 1713886819
    time_us : -2067971001
    time_done_s : 0
    time_done_us : 0
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 39 microseconds
Block 5 (Updating odb monitor) time: 4 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
read_trigger_event : Trigger info:
    trigger number 24572
    trigger mask 0
    time_s : 1713886819
    time_us : -2065393376
    time_done_s : 0
    time_done_us : 0
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 37 microseconds
Block 5 (Updating odb monitor) time: 4 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds

And if we turn verbose off, the printing goes to near zero:

Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 3 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 3 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 3 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 3 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds

However, the rate is limited to at best ~3kHz. 1/3us = 300kHZ, so there's something "hidden" going on.

Explicitly, these were the timings done:

INT read_trigger_event_with_timings(char *pevent, INT off) {
    // Define variables to store execution times
    long long totalTime1 = 0;
    long long totalTime2 = 0;
    long long totalTime3 = 0;
    long long totalTime4 = 0;
    long long totalTime5 = 0;
    long long totalTime6 = 0;

    // Block 1: Initialization
    auto start1 = std::chrono::high_resolution_clock::now();
    bk_init32(pevent);
    auto end1 = std::chrono::high_resolution_clock::now();
    totalTime1 = std::chrono::duration_cast<std::chrono::microseconds>(end1 - start1).count();

    // Block 2: Creating TRIG bank
    auto start2 = std::chrono::high_resolution_clock::now();
    DWORD *pdata;
    bk_create(pevent, "TRIG", TID_DWORD, (void**)&pdata);

    // Populate TRIG bank data
    *pdata++ = trigger_time_info.trigger_nr;
    *pdata++ = trigger_time_info.trigger_mask;
    *pdata++ = trigger_time_info.time_s;
    *pdata++ = trigger_time_info.time_us;
    *pdata++ = trigger_time_info.time_recv_s;
    *pdata++ = trigger_time_info.time_recv_us;
    *pdata++ = trigger_time_info.time_done_s;
    *pdata++ = trigger_time_info.time_done_us;

    bk_close(pevent, pdata);
    auto end2 = std::chrono::high_resolution_clock::now();
    totalTime2 = std::chrono::duration_cast<std::chrono::microseconds>(end2 - start2).count();

    // Block 3: Writing GPS timestamp
    auto start3 = std::chrono::high_resolution_clock::now();
    if (trigger_source == GPS) {
        DWORD *pdata;
        bk_create(pevent, "GPS0", TID_DWORD, (void**)&pdata);
        // Populate GPS0 bank data
        *pdata++ = gps_evnt_counter;
        *pdata++ = gps_tstamp_cap.sec;
        *pdata++ = gps_tstamp_cap.frac;
        *pdata++ = gps_tstamp_now.sec;
        *pdata++ = gps_tstamp_now.frac;

        bk_close(pevent, pdata);
    }
    auto end3 = std::chrono::high_resolution_clock::now();
    totalTime3 = std::chrono::duration_cast<std::chrono::microseconds>(end3 - start3).count();

    // Block 4: Printing trigger info (if verbose)
    auto start4 = std::chrono::high_resolution_clock::now();
    if (master_settings_odb.verbose) {
        printf("read_trigger_event : Trigger info:\n");
        printf("    trigger number %i\n", trigger_time_info.trigger_nr);
        printf("    trigger mask %i\n", trigger_time_info.trigger_mask);
        printf("    time_s : %i\n", trigger_time_info.time_s);
        printf("    time_us : %i\n", trigger_time_info.time_us);
        printf("    time_done_s : %i\n", trigger_time_info.time_done_s);
        printf("    time_done_us : %i\n", trigger_time_info.time_done_us);
    }
    auto end4 = std::chrono::high_resolution_clock::now();
    totalTime4 = std::chrono::duration_cast<std::chrono::microseconds>(end4 - start4).count();

    // Block 5: Updating odb monitor
    auto start5 = std::chrono::high_resolution_clock::now();
    int GPSFill = gps_evnt_counter;
    db_set_value(hDB, 0, "/Equipment/MasterGM2/Monitors/GPS Fill Number", &GPSFill, sizeof(GPSFill), 1, TID_INT);
    auto end5 = std::chrono::high_resolution_clock::now();
    totalTime5 = std::chrono::duration_cast<std::chrono::microseconds>(end5 - start5).count();

    // Block 6: Checking slave fills and triggering alarm
    auto start6 = std::chrono::high_resolution_clock::now();
    if (GPSFill % master_settings_odb.fill_number_check_interval == 0) {
        int count_threshold = 1;
        int slave_mismatched = check_slave_fills_odb_in_run(GPSFill, master_settings_odb.fill_number_alarm_threshold);

        if (slave_mismatched >= count_threshold && !FillNumberAlarmTriggered) {
            FillNumberAlarmTriggered = true;
            char AlarmMsg[500];
            sprintf(AlarmMsg, "DAQ | MasterGM2 discovered severe fill number mismatch");

            int ret_code = al_trigger_alarm("Fill Mismatch Error", AlarmMsg, "Alarm", "Fill Mismatch Error", AT_INTERNAL);
            if (ret_code != AL_SUCCESS) {
                cm_msg(MERROR, __FILE__, "Failure Raising Alarm: Error %d, Alarm \"%s\"", ret_code, "Fill Mismatch Error");
            }
        }
    }
    auto end6 = std::chrono::high_resolution_clock::now();
    totalTime6 = std::chrono::duration_cast<std::chrono::microseconds>(end6 - start6).count();

    // Print out execution times for each block
    printf("Block 1 (Initialization) time: %lld microseconds\n", totalTime1);
    printf("Block 2 (Creating TRIG bank) time: %lld microseconds\n", totalTime2);
    printf("Block 3 (Writing GPS timestamp) time: %lld microseconds\n", totalTime3);
    printf("Block 4 (Printing trigger info) time: %lld microseconds\n", totalTime4);
    printf("Block 5 (Updating odb monitor) time: %lld microseconds\n", totalTime5);
    printf("Block 6 (Checking slave fills and triggering alarm) time: %lld microseconds\n", totalTime6);

    return bk_size(pevent);
}
INT read_trigger_event_with_timings(char *pevent, INT off) {
    // Define variables to store execution times
    long long totalTime1 = 0;
    long long totalTime2 = 0;
    long long totalTime3 = 0;
    long long totalTime4 = 0;
    long long totalTime5 = 0;
    long long totalTime6 = 0;

    // Block 1: Initialization
    auto start1 = std::chrono::high_resolution_clock::now();
    bk_init32(pevent);
    auto end1 = std::chrono::high_resolution_clock::now();
    totalTime1 = std::chrono::duration_cast<std::chrono::microseconds>(end1 - start1).count();

    // Block 2: Creating TRIG bank
    auto start2 = std::chrono::high_resolution_clock::now();
    DWORD *pdata;
    bk_create(pevent, "TRIG", TID_DWORD, (void**)&pdata);

    // Populate TRIG bank data
    *pdata++ = trigger_time_info.trigger_nr;
    *pdata++ = trigger_time_info.trigger_mask;
    *pdata++ = trigger_time_info.time_s;
    *pdata++ = trigger_time_info.time_us;
    *pdata++ = trigger_time_info.time_recv_s;
    *pdata++ = trigger_time_info.time_recv_us;
    *pdata++ = trigger_time_info.time_done_s;
    *pdata++ = trigger_time_info.time_done_us;

    bk_close(pevent, pdata);
    auto end2 = std::chrono::high_resolution_clock::now();
    totalTime2 = std::chrono::duration_cast<std::chrono::microseconds>(end2 - start2).count();

    // Block 3: Writing GPS timestamp
    auto start3 = std::chrono::high_resolution_clock::now();
    if (trigger_source == GPS) {
        DWORD *pdata;
        bk_create(pevent, "GPS0", TID_DWORD, (void**)&pdata);
        // Populate GPS0 bank data
        *pdata++ = gps_evnt_counter;
        *pdata++ = gps_tstamp_cap.sec;
        *pdata++ = gps_tstamp_cap.frac;
        *pdata++ = gps_tstamp_now.sec;
        *pdata++ = gps_tstamp_now.frac;

        bk_close(pevent, pdata);
    }
    auto end3 = std::chrono::high_resolution_clock::now();
    totalTime3 = std::chrono::duration_cast<std::chrono::microseconds>(end3 - start3).count();

    // Block 4: Printing trigger info (if verbose)
    auto start4 = std::chrono::high_resolution_clock::now();
    if (master_settings_odb.verbose) {
        printf("read_trigger_event : Trigger info:\n");
        printf("    trigger number %i\n", trigger_time_info.trigger_nr);
        printf("    trigger mask %i\n", trigger_time_info.trigger_mask);
        printf("    time_s : %i\n", trigger_time_info.time_s);
        printf("    time_us : %i\n", trigger_time_info.time_us);
        printf("    time_done_s : %i\n", trigger_time_info.time_done_s);
        printf("    time_done_us : %i\n", trigger_time_info.time_done_us);
    }
    auto end4 = std::chrono::high_resolution_clock::now();
    totalTime4 = std::chrono::duration_cast<std::chrono::microseconds>(end4 - start4).count();

    // Block 5: Updating odb monitor
    auto start5 = std::chrono::high_resolution_clock::now();
    int GPSFill = gps_evnt_counter;
    db_set_value(hDB, 0, "/Equipment/MasterGM2/Monitors/GPS Fill Number", &GPSFill, sizeof(GPSFill), 1, TID_INT);
    auto end5 = std::chrono::high_resolution_clock::now();
    totalTime5 = std::chrono::duration_cast<std::chrono::microseconds>(end5 - start5).count();

    // Block 6: Checking slave fills and triggering alarm
    auto start6 = std::chrono::high_resolution_clock::now();
    if (GPSFill % master_settings_odb.fill_number_check_interval == 0) {
        int count_threshold = 1;
        int slave_mismatched = check_slave_fills_odb_in_run(GPSFill, master_settings_odb.fill_number_alarm_threshold);

        if (slave_mismatched >= count_threshold && !FillNumberAlarmTriggered) {
            FillNumberAlarmTriggered = true;
            char AlarmMsg[500];
            sprintf(AlarmMsg, "DAQ | MasterGM2 discovered severe fill number mismatch");

            int ret_code = al_trigger_alarm("Fill Mismatch Error", AlarmMsg, "Alarm", "Fill Mismatch Error", AT_INTERNAL);
            if (ret_code != AL_SUCCESS) {
                cm_msg(MERROR, __FILE__, "Failure Raising Alarm: Error %d, Alarm \"%s\"", ret_code, "Fill Mismatch Error");
            }
        }
    }
    auto end6 = std::chrono::high_resolution_clock::now();
    totalTime6 = std::chrono::duration_cast<std::chrono::microseconds>(end6 - start6).count();

    // Print out execution times for each block
    printf("Block 1 (Initialization) time: %lld microseconds\n", totalTime1);
    printf("Block 2 (Creating TRIG bank) time: %lld microseconds\n", totalTime2);
    printf("Block 3 (Writing GPS timestamp) time: %lld microseconds\n", totalTime3);
    printf("Block 4 (Printing trigger info) time: %lld microseconds\n", totalTime4);
    printf("Block 5 (Updating odb monitor) time: %lld microseconds\n", totalTime5);
    printf("Block 6 (Checking slave fills and triggering alarm) time: %lld microseconds\n", totalTime6);

    return bk_size(pevent);
}

23/04/2024 11:53

I noticed the "CCC Run aborted" alarm is raised at the start of a run if the AMC13001 is started before the master completely finishes initializing. Though, sometimes the "CCC Run aborted" alarm is raised even if we wait for the master to completely initialize before starting the AMC13001 frontend.

23/04/2024 11:56

I added this line to the top of the timings:

    // Print the timestamp when the function starts
    auto startTimestamp = std::chrono::system_clock::now();
    auto startTimestampMs = std::chrono::time_point_cast<std::chrono::milliseconds>(startTimestamp);
    auto timestamp = std::chrono::duration_cast<std::chrono::milliseconds>(startTimestampMs.time_since_epoch()).count();
    std::cout << "Function started at timestamp: " << timestamp << " ms" << std::endl;
    // Print the timestamp when the function starts
    auto startTimestamp = std::chrono::system_clock::now();
    auto startTimestampMs = std::chrono::time_point_cast<std::chrono::milliseconds>(startTimestamp);
    auto timestamp = std::chrono::duration_cast<std::chrono::milliseconds>(startTimestampMs.time_since_epoch()).count();
    std::cout << "Function started at timestamp: " << timestamp << " ms" << std::endl;

And the output:

Function started at timestamp: 1713888335077028 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 3 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Function started at timestamp: 1713888335077389 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 6 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Function started at timestamp: 1713888335077766 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 3 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Function started at timestamp: 1713888335078138 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 3 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Function started at timestamp: 1713888335078500 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 3 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Function started at timestamp: 1713888335078861 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Function started at timestamp: 1713888335077028 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 3 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Function started at timestamp: 1713888335077389 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 6 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Function started at timestamp: 1713888335077766 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 3 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Function started at timestamp: 1713888335078138 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 3 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Function started at timestamp: 1713888335078500 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds
Block 5 (Updating odb monitor) time: 3 microseconds
Block 6 (Checking slave fills and triggering alarm) time: 0 microseconds
Function started at timestamp: 1713888335078861 microseconds
Block 1 (Initialization) time: 0 microseconds
Block 2 (Creating TRIG bank) time: 0 microseconds
Block 3 (Writing GPS timestamp) time: 0 microseconds
Block 4 (Printing trigger info) time: 0 microseconds

For some reason then function calls are seperated by ~0.3ms (corresponds to us seeing max rate of 3kHz). However, the function itself isn't causing the lag.

23/04/2024 12:14

I tried to match the way CaloReadoutAMC13 and MasterGM2 frontends trigger, so I commented this line out:

//#define USE_INTERRUPT
//#define USE_INTERRUPT

So that MasterGM2's "Equipment" list uses the same equipment type as CaloReadoutAMC13:
MasterGM2/frontend.cpp:311

EQUIPMENT equipment[] = {
  { 
    "MasterGM2",                  /* equipment name */
    { // EQUIPMENT_INFO
      1,                          /* event ID */
      0xffff,                /* trigger mask */
      "BUF",                      /* event buffer */
#ifdef USE_INTERRUPT
      EQ_INTERRUPT | EQ_EB,               /* equipment type */
#else
      EQ_POLLED | EQ_EB,               /* equipment type */
#endif
      // LAM_SOURCE(0, 0xFFFFFF), /* event source crate 0, all stations */
      0,                          /* event source */
      "MIDAS",                    /* format */
      TRUE,                       /* enabled */
      RO_RUNNING ,                 /* read only when running */
      //| RO_ODB,                 /* and update ODB */
      1,                          /* poll for 1ms */
      0,                          /* stop run after this event limit */
      0,                          /* number of sub events */
      0,                          /* don't log history */
      "",                         /* frontend host */
      "",                         /* frontend name */
      "",                         /* source file  */
      "",                         /* current status of equipment */
      "",                         /* color to be used by mhttpd for status */
      FALSE                       /* hidden */
    },
    read_trigger_event,           /* pointer to readout routine */
  },
  {""}
};
EQUIPMENT equipment[] = {
  { 
    "MasterGM2",                  /* equipment name */
    { // EQUIPMENT_INFO
      1,                          /* event ID */
      0xffff,                /* trigger mask */
      "BUF",                      /* event buffer */
#ifdef USE_INTERRUPT
      EQ_INTERRUPT | EQ_EB,               /* equipment type */
#else
      EQ_POLLED | EQ_EB,               /* equipment type */
#endif
      // LAM_SOURCE(0, 0xFFFFFF), /* event source crate 0, all stations */
      0,                          /* event source */
      "MIDAS",                    /* format */
      TRUE,                       /* enabled */
      RO_RUNNING ,                 /* read only when running */
      //| RO_ODB,                 /* and update ODB */
      1,                          /* poll for 1ms */
      0,                          /* stop run after this event limit */
      0,                          /* number of sub events */
      0,                          /* don't log history */
      "",                         /* frontend host */
      "",                         /* frontend name */
      "",                         /* source file  */
      "",                         /* current status of equipment */
      "",                         /* color to be used by mhttpd for status */
      FALSE                       /* hidden */
    },
    read_trigger_event,           /* pointer to readout routine */
  },
  {""}
};

CaloReadoutAMC13/frontend.cpp:401

  EQUIPMENT equipment[] = { 
    {
      "AMC13%03d",              /* equipment name */
      {1, 0xffff,               /* event ID, trigger mask */
       "BUF%03d",               /* event buffer */
       EQ_POLLED | EQ_EB,       /* equipment type */
       LAM_SOURCE(0, 0xFFFFFF), /* event source crate 0, all stations */
       "MIDAS",                 /* format */
       TRUE,                    /* enabled */
       RO_RUNNING ,              /* read only when running and end of run */
       10,                      /* poll for 1ms */
       0,                       /* stop run after this event limit */
       0,                       /* number of sub events */
       0,                       /* don't log history */
       "", "", "",              /* frontend host, frontend name, frontend file name */
       "", "", FALSE},          /* status, status color, hidden */
      read_trigger_event,       /* readout routine */
    },
    {""}
  };
  EQUIPMENT equipment[] = { 
    {
      "AMC13%03d",              /* equipment name */
      {1, 0xffff,               /* event ID, trigger mask */
       "BUF%03d",               /* event buffer */
       EQ_POLLED | EQ_EB,       /* equipment type */
       LAM_SOURCE(0, 0xFFFFFF), /* event source crate 0, all stations */
       "MIDAS",                 /* format */
       TRUE,                    /* enabled */
       RO_RUNNING ,              /* read only when running and end of run */
       10,                      /* poll for 1ms */
       0,                       /* stop run after this event limit */
       0,                       /* number of sub events */
       0,                       /* don't log history */
       "", "", "",              /* frontend host, frontend name, frontend file name */
       "", "", FALSE},          /* status, status color, hidden */
      read_trigger_event,       /* readout routine */
    },
    {""}
  };

This sped the master up a little bit, but it still caps at ~3kHz.

25/04/2024 14:17

Crate errors when high trigger rate:
5db737920f631a39d8065712d10fc89a.png

25/04/2024 14:24

Truncated outpiut of mbggpscap:

[root@dhcp-10-163-105-238 ~]# mbggpscap

mbggpscap v4.2.24 copyright Meinberg 2001-2023

TCR180PEX 039212025430 (FW 1.21, ASIC 9.00) at port 0xE000, irq 45
Be sure the device has been properly configured to enable capture inputs.

On-board FIFO: 583 of 583 entries used.

Reading capture events:
New capture: CH0: 2024-04-25 18:18:28.8675413 UTC
New capture: CH0: 2024-04-25 18:18:28.8678414 UTC << BUF OVR
New capture: CH0: 2024-04-25 18:18:28.8684415 UTC
New capture: CH0: 2024-04-25 18:18:28.8687416 UTC
New capture: CH0: 2024-04-25 18:18:28.8690417 UTC << BUF OVR
New capture: CH0: 2024-04-25 18:18:28.8696418 UTC
New capture: CH0: 2024-04-25 18:18:28.8705421 UTC
New capture: CH0: 2024-04-25 18:18:28.8708421 UTC
New capture: CH0: 2024-04-25 18:18:28.8711422 UTC << BUF OVR
...
New capture: CH0: 2024-04-25 18:18:29.0832956 UTC << BUF OVR
New capture: CH0: 2024-04-25 18:18:29.0838957 UTC
New capture: CH0: 2024-04-25 18:18:29.0841958 UTC
New capture: CH0: 2024-04-25 18:18:29.0844958 UTC
New capture: CH0: 2024-04-25 18:18:29.0853961 UTC
New capture: CH0: 2024-04-25 18:18:29.0856961 UTC
New capture: CH0: 2024-04-25 18:18:29.0859962 UTC
New capture: CH0: 2024-04-25 18:18:29.0862963 UTC << BUF OVR
New capture: CH0: 2024-04-25 18:18:29.0868965 UTC
New capture: CH0: 2024-04-25 18:18:29.0871965 UTC
New capture: CH0: 2024-04-25 18:18:29.0877967 UTC << BUF OVR
New capture: CH0: 2024-04-25 18:18:29.0925979 UTC << BUF OVR
New capture: CH0: 2024-04-25 18:18:29.1154036 UTC
New capture: CH0: 2024-04-25 18:18:29.1157037 UTC
New capture: CH0: 2024-04-25 18:18:29.1160038 UTC
New capture: CH0: 2024-04-25 18:18:29.1163039 UTC
New capture: CH0: 2024-04-25 18:18:29.1166039 UTC
New capture: CH0: 2024-04-25 18:18:29.1172041 UTC
New capture: CH0: 2024-04-25 18:18:29.1175041 UTC
New capture: CH0: 2024-04-25 18:18:29.1178042 UTC
New capture: CH0: 2024-04-25 18:18:29.1181043 UTC
New capture: CH0: 2024-04-25 18:18:29.1184044 UTC
New capture: CH0: 2024-04-25 18:18:29.1187044 UTC
New capture: CH0: 2024-04-25 18:18:29.1193046 UTC
New capture: CH0: 2024-04-25 18:18:29.1196047 UTC
New capture: CH0: 2024-04-25 18:18:29.1199047 UTC
New capture: CH0: 2024-04-25 18:18:29.1202048 UTC
New capture: CH0: 2024-04-25 18:18:29.1205049 UTC
New capture: CH0: 2024-04-25 18:18:29.1208050 UTC
New capture: CH0: 2024-04-25 18:18:29.1214051 UTC
New capture: CH0: 2024-04-25 18:18:29.1217052 UTC
New capture: CH0: 2024-04-25 18:18:29.1220053 UTC
New capture: CH0: 2024-04-25 18:18:29.1223054 UTC
New capture: CH0: 2024-04-25 18:18:29.1226054 UTC
New capture: CH0: 2024-04-25 18:18:29.1229055 UTC
New capture: CH0: 2024-04-25 18:18:29.1235057 UTC
New capture: CH0: 2024-04-25 18:18:29.1238057 UTC
New capture: CH0: 2024-04-25 18:18:29.1241058 UTC
New capture: CH0: 2024-04-25 18:18:29.1244059 UTC
New capture: CH0: 2024-04-25 18:18:29.1247060 UTC
New capture: CH0: 2024-04-25 18:18:29.1250060 UTC
New capture: CH0: 2024-04-25 18:18:29.1256062 UTC
New capture: CH0: 2024-04-25 18:18:29.1259063 UTC
New capture: CH0: 2024-04-25 18:18:29.1262063 UTC
New capture: CH0: 2024-04-25 18:18:29.1265064 UTC
New capture: CH0: 2024-04-25 18:18:29.1268065 UTC
New capture: CH0: 2024-04-25 18:18:29.1271066 UTC
New capture: CH0: 2024-04-25 18:18:29.1277067 UTC
New capture: CH0: 2024-04-25 18:18:29.1280068 UTC
New capture: CH0: 2024-04-25 18:18:29.1283069 UTC
New capture: CH0: 2024-04-25 18:18:29.1286069 UTC
New capture: CH0: 2024-04-25 18:18:29.1289070 UTC
New capture: CH0: 2024-04-25 18:18:29.1292071 UTC
New capture: CH0: 2024-04-25 18:18:29.1295072 UTC
New capture: CH0: 2024-04-25 18:18:29.1301073 UTC
New capture: CH0: 2024-04-25 18:18:29.1304074 UTC
New capture: CH0: 2024-04-25 18:18:29.1307075 UTC
New capture: CH0: 2024-04-25 18:18:29.1310075 UTC
...
New capture: CH0: 2024-04-25 18:18:30.3712194 UTC
New capture: CH0: 2024-04-25 18:18:30.3790214 UTC
New capture: CH0: 2024-04-25 18:18:30.3793214 UTC
New capture: CH0: 2024-04-25 18:18:30.3796215 UTC
New capture: CH0: 2024-04-25 18:18:30.3799216 UTC
New capture: CH0: 2024-04-25 18:18:30.3802217 UTC
New capture: CH0: 2024-04-25 18:18:30.3805217 UTC
New capture: CH0: 2024-04-25 18:18:30.3808218 UTC
New capture: CH0: 2024-04-25 18:18:30.3811219 UTC
New capture: CH0: 2024-04-25 18:18:30.3817220 UTC
New capture: CH0: 2024-04-25 18:18:30.3820221 UTC
[root@dhcp-10-163-105-238 ~]# mbggpscap

mbggpscap v4.2.24 copyright Meinberg 2001-2023

TCR180PEX 039212025430 (FW 1.21, ASIC 9.00) at port 0xE000, irq 45
Be sure the device has been properly configured to enable capture inputs.

On-board FIFO: 583 of 583 entries used.

Reading capture events:
New capture: CH0: 2024-04-25 18:18:28.8675413 UTC
New capture: CH0: 2024-04-25 18:18:28.8678414 UTC << BUF OVR
New capture: CH0: 2024-04-25 18:18:28.8684415 UTC
New capture: CH0: 2024-04-25 18:18:28.8687416 UTC
New capture: CH0: 2024-04-25 18:18:28.8690417 UTC << BUF OVR
New capture: CH0: 2024-04-25 18:18:28.8696418 UTC
New capture: CH0: 2024-04-25 18:18:28.8705421 UTC
New capture: CH0: 2024-04-25 18:18:28.8708421 UTC
New capture: CH0: 2024-04-25 18:18:28.8711422 UTC << BUF OVR
...
New capture: CH0: 2024-04-25 18:18:29.0832956 UTC << BUF OVR
New capture: CH0: 2024-04-25 18:18:29.0838957 UTC
New capture: CH0: 2024-04-25 18:18:29.0841958 UTC
New capture: CH0: 2024-04-25 18:18:29.0844958 UTC
New capture: CH0: 2024-04-25 18:18:29.0853961 UTC
New capture: CH0: 2024-04-25 18:18:29.0856961 UTC
New capture: CH0: 2024-04-25 18:18:29.0859962 UTC
New capture: CH0: 2024-04-25 18:18:29.0862963 UTC << BUF OVR
New capture: CH0: 2024-04-25 18:18:29.0868965 UTC
New capture: CH0: 2024-04-25 18:18:29.0871965 UTC
New capture: CH0: 2024-04-25 18:18:29.0877967 UTC << BUF OVR
New capture: CH0: 2024-04-25 18:18:29.0925979 UTC << BUF OVR
New capture: CH0: 2024-04-25 18:18:29.1154036 UTC
New capture: CH0: 2024-04-25 18:18:29.1157037 UTC
New capture: CH0: 2024-04-25 18:18:29.1160038 UTC
New capture: CH0: 2024-04-25 18:18:29.1163039 UTC
New capture: CH0: 2024-04-25 18:18:29.1166039 UTC
New capture: CH0: 2024-04-25 18:18:29.1172041 UTC
New capture: CH0: 2024-04-25 18:18:29.1175041 UTC
New capture: CH0: 2024-04-25 18:18:29.1178042 UTC
New capture: CH0: 2024-04-25 18:18:29.1181043 UTC
New capture: CH0: 2024-04-25 18:18:29.1184044 UTC
New capture: CH0: 2024-04-25 18:18:29.1187044 UTC
New capture: CH0: 2024-04-25 18:18:29.1193046 UTC
New capture: CH0: 2024-04-25 18:18:29.1196047 UTC
New capture: CH0: 2024-04-25 18:18:29.1199047 UTC
New capture: CH0: 2024-04-25 18:18:29.1202048 UTC
New capture: CH0: 2024-04-25 18:18:29.1205049 UTC
New capture: CH0: 2024-04-25 18:18:29.1208050 UTC
New capture: CH0: 2024-04-25 18:18:29.1214051 UTC
New capture: CH0: 2024-04-25 18:18:29.1217052 UTC
New capture: CH0: 2024-04-25 18:18:29.1220053 UTC
New capture: CH0: 2024-04-25 18:18:29.1223054 UTC
New capture: CH0: 2024-04-25 18:18:29.1226054 UTC
New capture: CH0: 2024-04-25 18:18:29.1229055 UTC
New capture: CH0: 2024-04-25 18:18:29.1235057 UTC
New capture: CH0: 2024-04-25 18:18:29.1238057 UTC
New capture: CH0: 2024-04-25 18:18:29.1241058 UTC
New capture: CH0: 2024-04-25 18:18:29.1244059 UTC
New capture: CH0: 2024-04-25 18:18:29.1247060 UTC
New capture: CH0: 2024-04-25 18:18:29.1250060 UTC
New capture: CH0: 2024-04-25 18:18:29.1256062 UTC
New capture: CH0: 2024-04-25 18:18:29.1259063 UTC
New capture: CH0: 2024-04-25 18:18:29.1262063 UTC
New capture: CH0: 2024-04-25 18:18:29.1265064 UTC
New capture: CH0: 2024-04-25 18:18:29.1268065 UTC
New capture: CH0: 2024-04-25 18:18:29.1271066 UTC
New capture: CH0: 2024-04-25 18:18:29.1277067 UTC
New capture: CH0: 2024-04-25 18:18:29.1280068 UTC
New capture: CH0: 2024-04-25 18:18:29.1283069 UTC
New capture: CH0: 2024-04-25 18:18:29.1286069 UTC
New capture: CH0: 2024-04-25 18:18:29.1289070 UTC
New capture: CH0: 2024-04-25 18:18:29.1292071 UTC
New capture: CH0: 2024-04-25 18:18:29.1295072 UTC
New capture: CH0: 2024-04-25 18:18:29.1301073 UTC
New capture: CH0: 2024-04-25 18:18:29.1304074 UTC
New capture: CH0: 2024-04-25 18:18:29.1307075 UTC
New capture: CH0: 2024-04-25 18:18:29.1310075 UTC
...
New capture: CH0: 2024-04-25 18:18:30.3712194 UTC
New capture: CH0: 2024-04-25 18:18:30.3790214 UTC
New capture: CH0: 2024-04-25 18:18:30.3793214 UTC
New capture: CH0: 2024-04-25 18:18:30.3796215 UTC
New capture: CH0: 2024-04-25 18:18:30.3799216 UTC
New capture: CH0: 2024-04-25 18:18:30.3802217 UTC
New capture: CH0: 2024-04-25 18:18:30.3805217 UTC
New capture: CH0: 2024-04-25 18:18:30.3808218 UTC
New capture: CH0: 2024-04-25 18:18:30.3811219 UTC
New capture: CH0: 2024-04-25 18:18:30.3817220 UTC
New capture: CH0: 2024-04-25 18:18:30.3820221 UTC

It looks like the meinberg only "captures" events every ~300us ~ 3kHz (our max rate). Not sure what "BUF OVR" means, but it "goes away" after a little bit.

22/04/2024 02:12

I tried boosting the rate as high as I could with the internal trigger.
.
First I tried 4kHz. The Mater couldn't keep up (only can handle ~3Hz)
f65ed5beef75282e03cff45381fc8c78.png

read_trigger_event : New event, trigger # 253002
[MasterGM2,INFO] End of Run: DC7 Triggers Received 341722  Count triggers 253003
End-of-Run Trigger Number 341722
Frontend AMC13001, fill number 341722
[MasterGM2,INFO] End of Run: fills registered by all frontends match!
   done
Run stopped
read_trigger_event : New event, trigger # 253002
[MasterGM2,INFO] End of Run: DC7 Triggers Received 341722  Count triggers 253003
End-of-Run Trigger Number 341722
Frontend AMC13001, fill number 341722
[MasterGM2,INFO] End of Run: fills registered by all frontends match!
   done
Run stopped

I didn't reall understand why this was, so I got rid of some hardcoded printing around line 1535:

INT read_trigger_event(char *pevent, INT off)
{

  //printf("read_trigger_event : New event, trigger # %i\n",trigger_time_info.trigger_nr);

  DWORD *pdata;
INT read_trigger_event(char *pevent, INT off)
{

  //printf("read_trigger_event : New event, trigger # %i\n",trigger_time_info.trigger_nr);

  DWORD *pdata;

and ran make to remake the frontend.

This didn't seem to help. It seems the master still lags behind:
0edc7d886985981ab5e7be1e529c1a2e.png

Though, the DAQ doesn't crash for severl minutes. So I'm unsure what if this is intended in some way? Again, the master doesn't complain when a run is stopped.

CCC: Starting the run ...
Started run 83
EOR received: End of run 83
CCC: Stopping the run ...
TTS status at end of run
     CCC:  RDY (BUSY - 0.0 msec ago)

CCC: Check run state ...
CCC: Get trigger number ...
CCC: Update ODB fill number...
CCC: Waiting for meinberg trigger to match...
MasterGM2: Waiting to readout remaining triggers
[MasterGM2,INFO] End of Run: DC7 Triggers Received 432097  Count triggers 335829
End-of-Run Trigger Number 432097
Frontend AMC13001, fill number 432097
[MasterGM2,INFO] End of Run: fills registered by all frontends match!
   done
Run stopped
CCC: Starting the run ...
Started run 83
EOR received: End of run 83
CCC: Stopping the run ...
TTS status at end of run
     CCC:  RDY (BUSY - 0.0 msec ago)

CCC: Check run state ...
CCC: Get trigger number ...
CCC: Update ODB fill number...
CCC: Waiting for meinberg trigger to match...
MasterGM2: Waiting to readout remaining triggers
[MasterGM2,INFO] End of Run: DC7 Triggers Received 432097  Count triggers 335829
End-of-Run Trigger Number 432097
Frontend AMC13001, fill number 432097
[MasterGM2,INFO] End of Run: fills registered by all frontends match!
   done
Run stopped

22/04/2024 02:23

Trying to set the the internal trigger rate to 1/150us =~6.6kHz immediately caused CCC run abort. It's unclear why.

22/04/2024 02:29

At 5kHz, I was able to run for about 30 seconds, before getting CCC error. I see the Master is still stuck at about the same rate.
a2ca30bb5d3bf83c2ce932c2577b5a24.png

In short, I can't reproduce the software lagging behind with the internal trigger.

The hardware is giving these errors in the crate monitor:
b60912ea946121061aabd93140d0723c.png
7d4a760abf5fe4140525d40f12c93132.png

22/04/2024 02:49

I was able to run at 3KHz with the logger for ~15 seconds. I turned it off because large data rates were made. We can test this with fewer digitizers as well.

22/04/2024 16:03

Turning debug on and running at 4kHz and 5kHz, I do actually see ReadXBytes returning -1:

ReadXBytes(875): ReadXBytes :: x = 8312
read_trigger_event(3255): made Rider header / trailer databank CB001 size (bytes) 0x00000618, rider[0] 0x f0400009b030002, readout electronics fill number 1713816110
CZ bank,  Fill_type 1, frontend_index 1
ReadXBytes: warning read return code -1, errno 104
read_trigger_event(3286): made trailer databank CZ001 size 0x00000008, tail[0] 0x    c7dd5b1400a0, readout electronics fill number 1713816110
read_trigger_event(3320): lossless data compression 0
read_trigger_event(3477): lossless compression and bank deletion duration: dt = 0 s 6 us
CC bank,  Fill_type 1, frontend_index 1
read_trigger_event(3511): made timing databank CC001 size (bytes) 0x000000b0, amc13[0] 0x 800a01f9b030051, readout electronics fill number 1713816110
read_trigger_event(3561): tcp got header to tcp got data duration: dt = 0 s 680 us
read_trigger_event(3562): tcp got data to MFE done duration: dt = 0 s 493 us
read_trigger_event(3563): tcp got header to MFE done duration: dt = 0 s 1173 us
read_trigger_event(3564): gpu done to MFE done  duration: dt = 1713816110 s 291121 us
read_trigger_event(3565): midas bank size: 43492
[AMC13001,ERROR] [tcp_thread.cxx:893:tcp_thread.cxx,ERROR] ReadXBytes: warning read return code -1, errno 104
[AMC13001,ERROR] [tcp_thread.cxx:1178:tcp_thread.cxx,ERROR] Error when reading from socket, fd 40. Read -1 bytes vs 8312
[AMC13001,ERROR] [tcp_thread.cxx:655:tcp_thread.cxx,ERROR] tcp_thread: break the tcp thread loop becuase of a reading error -1
ReadXBytes(875): ReadXBytes :: x = 8312
read_trigger_event(3255): made Rider header / trailer databank CB001 size (bytes) 0x00000618, rider[0] 0x f0400009b030002, readout electronics fill number 1713816110
CZ bank,  Fill_type 1, frontend_index 1
ReadXBytes: warning read return code -1, errno 104
read_trigger_event(3286): made trailer databank CZ001 size 0x00000008, tail[0] 0x    c7dd5b1400a0, readout electronics fill number 1713816110
read_trigger_event(3320): lossless data compression 0
read_trigger_event(3477): lossless compression and bank deletion duration: dt = 0 s 6 us
CC bank,  Fill_type 1, frontend_index 1
read_trigger_event(3511): made timing databank CC001 size (bytes) 0x000000b0, amc13[0] 0x 800a01f9b030051, readout electronics fill number 1713816110
read_trigger_event(3561): tcp got header to tcp got data duration: dt = 0 s 680 us
read_trigger_event(3562): tcp got data to MFE done duration: dt = 0 s 493 us
read_trigger_event(3563): tcp got header to MFE done duration: dt = 0 s 1173 us
read_trigger_event(3564): gpu done to MFE done  duration: dt = 1713816110 s 291121 us
read_trigger_event(3565): midas bank size: 43492
[AMC13001,ERROR] [tcp_thread.cxx:893:tcp_thread.cxx,ERROR] ReadXBytes: warning read return code -1, errno 104
[AMC13001,ERROR] [tcp_thread.cxx:1178:tcp_thread.cxx,ERROR] Error when reading from socket, fd 40. Read -1 bytes vs 8312
[AMC13001,ERROR] [tcp_thread.cxx:655:tcp_thread.cxx,ERROR] tcp_thread: break the tcp thread loop becuase of a reading error -1

turning debug print off, it stops immediately crashing at 4kHz and 5kHz. The printout, even when screened, causes issues.

22/04/2024 16:16

When running at 9kHz, I quickly run into this issue:

17:09:53.604 2024/04/22 [mhttpd,INFO] Run #95 stopped

17:09:53.583 2024/04/22 [AMC13001,ERROR] [frontend.cpp:2457:frontend.cpp,ERROR] TCP/GPU/Midas fill numbers do not match at the end of the run.

17:09:53.583 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:53.583 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:53.583 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:53.583 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:35.430 2024/04/22 [MasterGM2,TALK] Warning: DAQ | Suspect fill number mismatch. Check Event numbers!

17:09:35.430 2024/04/22 [MasterGM2,ERROR] [frontend.cpp:1273:frontend.cpp,ERROR] End of Run: checking other frontend fill numbers, time out!

17:09:35.430 2024/04/22 [MasterGM2,INFO] End of Run: DC7 Triggers Received 78944  Count triggers 27291

17:09:35.430 2024/04/22 [MasterGM2,ERROR] [frontend.cpp:1194:end_of_run,ERROR] FC7-10: Unable to Verify Run has Stopped (Run state still in progress)

17:09:32.005 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:27.011 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:22.006 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:17.012 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:12.007 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:07.002 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:02.008 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:57.003 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:52.009 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:47.004 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:42.010 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:37.005 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:32.011 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:27.006 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:22.012 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:17.007 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:12.002 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:07.008 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:02.344 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:02.344 2024/04/22 [AMC13001,TALK] Warning: DAQ | AMC13001 TCP Ring buffer close to full (100.000000%)

17:07:53.591 2024/04/22 [mhttpd,INFO] Run #95 started
17:09:53.604 2024/04/22 [mhttpd,INFO] Run #95 stopped

17:09:53.583 2024/04/22 [AMC13001,ERROR] [frontend.cpp:2457:frontend.cpp,ERROR] TCP/GPU/Midas fill numbers do not match at the end of the run.

17:09:53.583 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:53.583 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:53.583 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:53.583 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:35.430 2024/04/22 [MasterGM2,TALK] Warning: DAQ | Suspect fill number mismatch. Check Event numbers!

17:09:35.430 2024/04/22 [MasterGM2,ERROR] [frontend.cpp:1273:frontend.cpp,ERROR] End of Run: checking other frontend fill numbers, time out!

17:09:35.430 2024/04/22 [MasterGM2,INFO] End of Run: DC7 Triggers Received 78944  Count triggers 27291

17:09:35.430 2024/04/22 [MasterGM2,ERROR] [frontend.cpp:1194:end_of_run,ERROR] FC7-10: Unable to Verify Run has Stopped (Run state still in progress)

17:09:32.005 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:27.011 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:22.006 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:17.012 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:12.007 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:07.002 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:09:02.008 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:57.003 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:52.009 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:47.004 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:42.010 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:37.005 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:32.011 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:27.006 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:22.012 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:17.007 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:12.002 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:07.008 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:02.344 2024/04/22 [AMC13001,INFO] Requesting Encoder FC7 to throttle TTC triggers to clear TCP/GPU ring buffers.

17:08:02.344 2024/04/22 [AMC13001,TALK] Warning: DAQ | AMC13001 TCP Ring buffer close to full (100.000000%)

17:07:53.591 2024/04/22 [mhttpd,INFO] Run #95 started

22/04/2024 17:35

It looks like the CCC error at the beginning of a run is indepedent of rate. It seems to occur almost randomly. Just restarting both the frontend seems to fix the issue.